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ABSTRACT 

A study assessed the practical use of the simplified 
speech pattern approach to teaching lipreading in a tone language by 
comparing performance using an acoustic hearing-aid and a Sivo-aid in 
a tone labelling task. After initial assessment, subjects were given 
training to enhance perception of lexically contrastive tones, then 
post-tested. The test used was audio-video recorded test material in 
Cantonese used to assess speech receptive ability, which includes a 
tonal component. For each subject „ visual stimuli were presented, 
some accompanied by an auditory signal (either speech tone or larynx 
tone) through either acoustic hearing aids or a Sivo-aid. one 
treatment used lipreading alone. Subjects in the experimental group 
were foui profoundly deaf children; all of their deaf classmates 
served as a control group. Results indicated that the auditory signal 
contributes to tonal reception more effectively using the Sivo-aid, 
but some variation still occurred. It is concluded that the 
simplification approach allows a better match to the user's hearing 
capabilities, makirg better use o; residual hearing, and allows 
gradual development of perceptual skills relating to suprasegmentals 
as well as segmentals. (MSE) 
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TONES FOR PROFOUNDLY DEAF TONE • LANGUAGE SPEAKERS * 

Teresa Ching 
CUHK 

X significance of tones 

The use of tones to signal different meanings is an 
inherent characteristic of a tone language like 
Cantonese. Syllable-based tonal contrasts can be 
realized by one of six tonal contours, namely, the high 
level tone, the high rising tone, the mid level tone, the 
low falling tone, the low rising tone, and the low level 
tone. This may be illustrated by the segmental 
combination /ji/. The utterance means "clothing", 
"chair", "meaning", "child", "ears" or "two", according 
to which of the six tones the speaker has selected in his 
production of /ji/. This systematic use of pitch 
variations on the syllable level constitutes lexical tone 
contrasts. Tonal patterns can also be superimposed on an 
utterance for discoursal functions, such as interrogative 
as opposed to declarative; and for affective purposes, 
for instance, to express attitudes and emotions. This 
linguistic use of pitch on the sentence level is 
intonation, and is also characteristic of non-tone 
languages, such as English (see, for example, 0* Connor 
and Arnold, 1965) . 

The production of tonal contrasts relates primarily 
to the speaker's control of the rate of vibration of his 
vocal folds in phonation. The hearer perceives changes 
in the fundamental frequency as differences in pitch. 
When speech communication takes place in a tone language 
environment, the ability to produce and perceive tonal 
contrasts is of utmost importance, for lexical 
distinctions and grammatical meanings are dependent on 
the systematic use of voice pitch. Figure 1 shows the 
contrastive tone patterns in Cantonese. 
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been reported that profoundly hearing* impaired listeners 
are best at discriminating frequency changes at low 
frequencies, particularly when these are presented as a 
pure tone rather than a complex tone (Stock and Rosen, 
1986) . The limited auditory functioning of the impaired 
ear is not well served by the conventional acoustic 
hearing-aids used. Lebisch (1989), for instance, 
commented on the mismatch between the recommendation of 
the audiologist who fits the hearing-aids and the 
preference of the experienced user. The latter 
consistently adjusts for a higher gain in the low 
frequency end, and a lesser gain above 1kHz than the 
settings suggested by the professional. As frequency 
selectivity worsens (widened critical bands) and 
frequency discriminability degrades at higher 
frequencies, what can be processed may be masked by other 
components in the amplified signal. Background noise 
would also exert influence on the entire auditory 
spectrum. Not only is the deaf listener unable to use 
the additional information in the speech signal, he is 
also deprived of what he can use. 

Even in the case of profound impairment, there is 
usually some residual hearing abilities. For example, 
Rosen et al (1987) reported for their profoundly deaf 
listeners the ability to distinguish periodic from 
aperiodic sounds, and to discriminate changes in 
frequency in the low frequency range (below 500Hz) . 
These capabilities are of practical importance in speech 
perception. 

The range within which there is residual hearing 
corresponds to the larynx frequency range of the speaker. 
Figure 2 shows a typical left-hand corner audiogram of a 
profoundly deaf listener, and the corresponding larynx 
frequency range of two Cantonese speakers. 



Figure 2 about here 



Variations in tone and intonation are conveyed by 
systematic changes in the fundamental frequency patterns 
in the speaker's voice, and these occur below 500Hz in 
normal speech. If the varying voice pitch could be 
clearly conveyed to the profoundly deaf listener, he 
would be able to discriminate changes in frequency in 
this range and use the tonal information for 
understanding speech. 



III Speech pattern approach to enhancing tonal 

perception 

Tone patterns are most useful to the impaired 
listener who has to depend on lipreading for 
understanding speech. For these patterns to be presented 
in a way matched to the limited auditory abilities, 
simplification is necessary. Speech pattern features can 
be extracted from the complex speech signal and presented 
in a simplified form. 

The extraction of tonal patterns from speech can be 
carried out by the use of a laryngograph, which senses 
vocal fold closure by two electrodes externally placed on 
the speaker's neck (Fourcin, 1974). Moore et al (1985) 
reported extensive work using the speech pattern 
extraction approach. Fourcin (1986a) described the 
possibility of presenting simplified tonal information by 
means of a specially designed hearing-aid. A profoundly 
deaf listener using the Sivo-aid, which provides him only 
with a sinusoidal representation of voice fundamental 
frequency variation, performs consistently better at 
lipreading connected speech than when he uses his own 
high quality acoustic hearing-aid, which provides him 
with the entire speech signal. Rosen et al (1987) also 
provided evidence showing that for some impaired 
listeners, simplifying the speech signal to a sinusoid is 
advantageous to speech reception. 

The presentation of tonal and prosodic information 
in a simplified form can be implemented in the use of the 
Sivo-aid. The device basically receives a speech input, 
extracts the voice fundamental frequency from the signal, 
and generates an acoustic sinusoid at the appropriate 
frequency and amplitude level. The implementation is 
aetailed in Rosen et al, 1987. 

IV video-recorded tests of tone perception 

Since visual information is of crucial importance to 
speech perception, particularly for hearing-aid users, 
voice pitch information must be evaluated as an aid to 
lipreading. The relative usefulness of presenting a 
simplified speech pattern for enhancing speech 
perception is assessed using video-recorded 
lipreading tests. It is profitable to start with simple 
patterns of contrasts and to gradually build up in 
complexity on the basis of increasing speech processing 
abilitier. 

The present work aims at assessing the practical use 
of the simplified speech pattern approach by comparing 
the performance using an acoustic hearing-aid and the 
Sivo-aid in a tone labelling task. The initial 
assessment is complemented by a related programme of 
training to enhance perception of lexically contrastive 




tones. Subsequent assessment provides an objective 
measure of progress. 

1 Method 

A battery of audio-video recorded test material in 
Cantonese has been developed to assess speech receptive 
ability. This includes a tonal component and a 
consonantal component. 

Of primary concern here is the tests on tonal 
contrasts. These are graded, from two tone 
discrimination involving the High Level and the Low 
Falling; to three tone contrasts involving the High Level 
vs. High Rising vs. Low Falling; to the entire system of 
six tone contrast, namely High Level, High Rising, Mid 
Level, Low Falling, Low Rising, and Low Level tones. 
These tonal contrasts are all based on /ji/, which gives 
readily available lexical labels corresponding to the six 
tonal patterns (see Figure 1 for the larynx frequency 
patterns) . 

A female native speaker of Cantonese was 
recorded. Figure 3 shows the voice characteristics of 
the speaker. 



Figure 3 about here 



The frequency of vocal fold vibration was derived from a 
long term sample of the larynx waveform, determined on a 
period by period basis. The second order probability 
histogram shows the distribution of laryngeal periods at 
different frequencies. The modal frequency is 243Hz, 
with a wide range of vibratory activity in the dominant 
range of 170 - 329Hz. This range within which varied 
tonal and intonational patterns are realized can be 
illustrated in the digram distribution, derived by 
plotting successive periods against each other. (For a 
detailed explanation of this laryngograph-based micro- 
computer analysis of vocal fold function, see Fourcin, 
1986b) . 

The recordings were made on a Sony U~matic high band 
video-cassette-recorder. Speech was recorded on one 
channel and simultaneously derived laryngograph signal on 
the other. 

Each of the test items was embedded in a frame 
/"ni-ko -hai / (this is ), the inter- 

stimuli interval was 7 seconds. Each test consisted of 



ten tokens of each stimulus, Baking a total of twenty in 
the two-tone tests, and thirty in the three-tone tests. 
A practice session preceded each test session, giving the 
fundamental frequency patterns of the contrast ive tones 
and the corresponding lexical items. 

Response sheets were used, on which the subjects 
were required to circle the appropriate lexical item 
after each stimulus was presented. 

2 Conditions 

All conditions used visual stimuli, and four of them 
presented auditory signal either via sound field when the 
subjects used their own hearing-aids or through the 
Sivo-aid. The auditory signal can be one of two types, 
speech tones or larynx tones. A fifth condition was 
lipreading alone. 

The tests aimed at assessing the relative 
contribution of the conventional hearing-aid and the 
Sivo-aid; and the use of speech tones and larynx tones 
for lexical tone reception. The Sivo-aid presents 
information related only to the voice fundamental 
frequency of the speaker, with practically no spectral 
information; while the conventional hearing-aid presents 
the complex speech signal. The speech tones contain 
vocal tract information, while the larynx tones contain 
basically tonal information. 

Before the testing session, each subject adjusted 
for the most comfortable listening level for all 
conditions with auditory signal. The individual settings 
for each subject when listening to recorded speech and 
recorded larynx tones were also observed for optimal 
reception using the Sivo-aid. 

3 Subjects 

Four profoundly deaf children served as subjects 
(Figure 4 gives the pure tone audiograms of the 
children). After the initial assessment, the children 
were placed in weekly training sessions using the Sivo- 
aid and a visual display (a Visi-pitch display) . The 
subjects were tested again after a four-week programme. 
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A control group, which comprised all students in the 
same class as the subjects, was also tested. They all 



used their own hearing-aids and lipread speech tones, 
larynx tones, and finally lipread with no auditory 
signal. All children were in the age range of 10-12 
years . 

Ml subjects were tested individually in a sound- 
treated room. The video signal was presented on a 12 in. 
colour monitor and the audio signal via a loudspeaker for 
the sound field conditions. As for the Sivo-aid 
conditions, the audio signal was fed directly into the 
hearing-aid. 

4 Results 

The two-tone labelling tests were used. This 
involves labelling a stimulus as meaning either 
"clothing" (for the high level tone) or "child" (for the 
low falling tone) . The orthographic representations of 
these lexical labels are well within the knowledge of 
even the youngest child tested. 

Table 1 gives the labelling results of all children. 
Responses were generally random (50% or below), showing 
no discrimination between the two tones. Either the 
subjects did not understand the task completely, or they 
lack the ability to classify the simple stimuli 
accurately. it is apparent that tonal contrasts cannot 
be lipread, even in a closed class of two oppositions. 
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Four of the twenty-three children tested attained a 
criterion level of 75% correct in the two-tone tests when 
lipreading speech tones with their own hearing-aids. 
These are severely hearing-impaired children with an 
average hearing loss in the region 75 - 90dB over the 
first four frequencies. They are making better use of 
their residual hearing than their profoundly deaf peer 
group. They are also getting additional information from 
the speech tones than from the larynx tones. 

Further assessment using a three-tone test and 
subsequent tests of the entire tonal system would reveal 
limitations in their linguistic capabilities, which could 
then be complemented by a rehabilitative programme to 
improve their prosodic skills. Normally hearing children 
performing the same task can confidently label the two 
tone contrasts accurately by five, and the three tone 
contrasts by six. Ching (1984) reported that children 
made confident judgments on the entire system of tonal 



contrasts in Cantonese by about ten years of age. 

As the present work is concerned with using a 
simplified approach to enhance tone receptive ability 
in the profoundly deaf, the performance of the 
profoundly deaf subjects is of primary interest. 

Comparing the four subjects* performance across 
conditions of presentation, it can be seen that the 
auditory signal contributes to tonal reception via the 
Sivo-aid more effectively (see Table 2). Subject 2. and 
subject 2 performed better when using the Sivo-aid than 
when using their own aid. Subject 3 responded best to 
the larynx tones received via the Sivo-aid. Subject 4 
was not able to identify the high level and low falling 
tones at all. 
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The Sivo-aid presented an auditory signal 
corresponding to the fundamental frequency of the 
utterance in the form of a sinusoid. The signal is well 
defined in periodicity, and frequency changes can be 
clearly conveyed. This accounts for the significantly 
higher scores in the Sivo-aid conditions as compared with 
those in the Own aid conditions for subject 1. He makes 
no use of the spectral information presented through his 
own hearing-aid. It is likely that the complex signal 
precludes the effective use of the available fundamental 
frequency information. The loss of acoustic information 
in the sinusoidal presentations is not associated with a 
potential loss in labelling efficiency. The scores for 
the Sivo-aid conditions are not significantly different 
from those in the Own aid conditions for the other three 
subjects. In fact, the scores are all slightly higher. 

The qeneral inability in lexical tone labelling can 
thus be attributed no„ so much to the loss of auditory 
information, but rather to the lack of listening 
experience to classify the auditory information 
meaningfully and accurately. 

5 Training 

Extensive training can make a lot of difference in 
speech perceptual tasks. White et al (1982), for 
instance, used a question/statement test to assess the 
perception of fundamental frequency contours of a patient 
using different speech coding schemes. Higher scores 
were obtained with the scheme which presented fundamental 
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frequency of the utterance only, but the score was not 
significantly different from chance.' Fourcin et al 
(1979) , however, reported significant improvement on the 
sane task for their four patients using an auditory 
stimulation which presented only fundamental frequency. 
The patients reported in the latter study had had 
considerable training in labelling prosodic contrasts 
before they were tested. 

While sinusoidal presentations can simplify the 
acoustic signal so that it is better matched to the 
limited sensory capability of the profoundly deaf 
listener, appropriate training can help in the 
development of the perception of tonal contrasts, so that 
it can be an important source of information for the 
lipreader. 

The four subjects were given extensive training 
specifically on the tonal contrasts in Cantonese. They 
used the .Sivo-aid for each training session so that the 
larynx frequency patterns of the speaker could be clearly 
presented auditorily. There was also a visual display to 
help them monitor their own production. Live training in 
perceiving and producing the tonal contrasts lasted 
fifteen minutes per session, and three sessions were 
given per week to each individual subject. The training 
programme initially went on for four weeks, after which 
the subjects were tested again. 

Figure 5 compares the scores before and after 
training for the four subjects using the video-recorded 
lipreading tests, showing overall improvement with 
training. Table 2 shows the relative enhancement of 
labelling ability for each individual subject. 
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Subject 1 showed confident labelling and performed 
perfectly when the larynx tones were presented via the 
Sivo-aid. He also scored 75% correct when using his own 
aid. There is learning effect for all lipreading 
conditions on the same task. Subject 2 demonstrated 
significant improvement, scoring 85% correct labelling 
when using her own aid and 90% correct using the Sivo-aid 
after training. Obviously she was not only gaining from 
the simplified signal but the increased listening 
experience also enabled her to make better use of the 
acoustic signal presented via her own aid. The results 
of subject 3 before and after training were not very much 
different. Although he was able to attend to the 
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differences between the two tones in live training, his 
labelling was not very confident. Further training is 
necessary so that he can better Mentify the perceived 
differences. Subject 4 is probably poorest in frequency 
resolution ability, although the pure tone audiogram does 
not give information relating to this aspect. His 
labelling score was 90% correct when larynx tones were 
presented via the Sivo-aid. The speech tones, which 
contain more spectral information possibly interferes 
with his effective use of the acoustic signal for 
identifying the contrasts between a high level tone and a 
low falling tone. While he was completely unable to 
classify the stimuli before training, he scored 60% 
correct when lipreading with auditory speech tones after 
training. 

6 Further work 

On the basis of the improved labelling ability of 
subjects 1 and 2, a third tone, the high rising tone was 
introduced. The subjects were first assessed using the 
video-recorded three tone lipreading tests. Subject 1 
labelled the high level and low falling tones 
confidently, but was completely unable to identify the 
new tone. Subject 2 showed degraded performance and 
overall inability to classify the stimuli. The subjects 
were then given appropriate training on the three tone 
contrasts in Cantonese, using the Sivo-aid and the visual 
display. The subjects were re-assessed after ten 
training sessions. 

Figure 6 compares the subjects' performance before 
and after training. Subject 1 showed overall 
improvement, and perfect identification when lipreading 
with larynx tones presented via the Sivo-aid. He also 
scored 83% correct using his own aid when listening to 
speech tones. Subject 2 scored 60% correct for the Sivo- 
aid condition, but her overall performance using her own 
aid was not better than chance. 
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V Summary and Discussion 

The present work uses a simplified speech pattern 
approach to enhance perception of tonal contrasts in 
Cantonese. The speech pattern is directly related to the 
larynx frequency of the speaker, which is known to aid 
lipreading most; and the simplification is implemented in 
the use of the Sivo-aid. Lipreading tests designed on 
the basis of speech patterns and graded in terms of 
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complexity are useful for identifying the difficulties, 
guiding specific training, and measuring learning 
effects. Appropriate training has led profoundly deaf 
subjects from a complete inability to discriminate tones 
to confident labelling and identification of lexical 
tones . 

As speech pattern features may gradually increase in 
complexity to encompass more contrast ive information, it 
would be possible to help profoundly deaf listeners to 
develop perceptual skills relating to supra segmental as 
well as segmental contrasts in speech. 

The simplification approach makes it possible to 
provide a better match to the hearing capabilities of the 
user, so that he could mak'i better use of his residual 
hearing. While speech pattern features relating to 
periodicity can be better carried by sinusoidal 
presentations, the range of frequency variation can also 
be transposed to a region at which there is optimal 
discriminative ability. So long as the essential 
patterns are preserved, the information thus presented 
can be usefully employed for speech perception. 

Especially when working in a tone language 
environment, hearing-aiU design or stimulation techniques 
need to take into consideration the potential benefits 
obtainable with the profoundly deaf listener using a 
speech pattern extraction and simplification approach. 
Assessment and rehabilitation work guided by this 
knowledge would speed up learning process and ensure 
meaningful results, both with regard to the development 
of perceptual skills as well as adequate expressive 
skills. 
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Fx PATTERNS OF TONES 




CANTONESE TONES 

1 HIGH LEVEL 4 LOW FALLING 

2 HIGH RISING 5 LOW RISING 

3 MID LEVEL 6 LOW LEVEL 



Figure 1 gives the fundamental frequency patterns (measured on 
a period-by-period basis) of the Cantonese tones 
produced by a male and a female speaker. The 
relative patterning of the tonal contours is the 
same, although there are absolute differences in 
frequency . 
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Figure 2 shows a typical left-hand-corner pure tone audiogram 
and the fundamental frequency distribution of two 
adult Cantonese speakers, a male and a female. The 
frequency range at which there is measurable hearing 
coincides with that of voice pitch variations in 
speech. 
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Figure 3 presents the fundamental frequency distribution of 
the speaker used for producing the tonal stimuli. 
Her range is indicated by the scatter plot* 
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Figure 4 gives the pure tone audiograms of the four profoundly 
deaf children who served as subjects ♦ 
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Figures shows the labelling of the four subjects 

"P read «9 with their own aids (SF) 
Jw!!2« ng no audit °ry stimuli (LA) and 

lipreading with the Sivo-aid (sivo) . ' The 
auditory stimulus, when present, it 
either speech (Sp) or larynx tones (Lx) 

K r V r ° a n ini 0 i C ° rrect ^belling befor e ( 2nd 
after training is compared for the subjects 
Two-tone contrasts were used as stimuli 
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Figure 6 compares the labelling performance of two 
subjects before and after training for the 
five conditions. 
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LIPREADING CANTONESE TONES 
HIGH LEVEL : LOW FALLING 



Conditions Lips Lips Lips 

+ + 
Speech Larynx tones Alone 



Proportion correct (%) 
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Subjects 
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70 
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45 
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30 


35 
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60 


40 


25 
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50 


50 


50 



Table 1 gives the results cf the two tone labelling 
tests. Twenty-three children took the 
tests, and they lipread under three 
conditions: lipreading with speech tones, 
lipreading with larynx tones, and 
lipreading alone. The children used their 
own aids in conditions with auditory stimuli. 
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LIPREADING CANTONESE TORIES 
HIGH LEVEL : LOW FALLING 



Conditions 

Own aid Own aid Lips Sivo aid Sivo aid 

+ + + + 
Speech Larynx tones Alone Speech Larynx tones 

Proportion correct 

SI (Before Training) 

14/20 9/20 9/20 16/20 15/20 

(After Training) 

15/20 12/20 12/20 17/20 20/20 



S2 (Before Training) 

12/20 6/20 7/20 13/20 8/20 

(After Training) 

17/20 13/20 7/20 18/20 14/20 



S3 (Before Training) 

12/20 11/20 5/20 6/20 13/20 

(After Training) 

12/20 11/20 8/20 10/20 13/20 



S4 (Before Training) 

10/20 10/20 10/20 9/20 10/20 

(After Training) 

12/20 7/20 10/20 12/20 18/20 



Table 2 Individual labelling scores of four subjects 
before and after training for the five 
conditions in a two-tone lipreading test. 
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